Image generation apparatus and image generation method

ABSTRACT

Methods and apparatus provide for an acquisition process to acquire information relating to at least one of a position and a rotation of a camera and an image generation process to generate an image to be displayed on a display unit in a first frequency lower than a second frequency corresponding to a frame rate of the display unit using information relating to at least one of a position and a rotation acquired at a certain point of time by the acquisition process.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of U.S. patent application Ser. No.:17/237,523, allowed, accorded a filing date of Apr. 22, 2021; which is acontinuation application of U.S. patent application Ser. No.:16/867,716, accorded a filing date of May 6, 2020 (U.S. Pat. No.(11,016,297 issued May 25, 2021); which is a continuation application ofU.S. patent application Ser. No.: 16/160,612, accorded a filing date ofOct. 15, 2018 (U.S. Pat. No. 10,684,475 issued Jun. 16, 2020); which isa continuation application of U.S. patent application Ser. No.:15/033,701, accorded a 371(c) date of May 2, 2016 (U.S. Pat. No.10,133,073 issued Nov. 20, 2018); which is a national stage applicationof PCT/JP2014/079082, filed Oct. 31, 2014; which claims priority to JP2013-233428, filed November 11, 2013,

the entire disclosures of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to an apparatus for and a method ofgenerating and correcting an image.

BACKGROUND ART

A user of a head-mounted display unit connected to a game machine wouldwear the head-mounted display unit on the head and operate a controlleror the like to play a game while watching a screen image displayed onthe head-mounted display unit. In an ordinary display unit of theinstallation type connected to a game machine, the visual field range ofthe user spreads also to the outside of the screen image of the displayunit. Therefore, the user cannot sometimes concentrate its attention tothe screen image of the display unit or sometimes lacks in sense ofimmersion in the game. In this regard, if the head-mounted display unitis worn, then the user does not look at any other than the imagedisplayed on the head-mounted display unit. Therefore, there is aneffect that the sense of immersion in the video world is enhanced andthe entertainment property of the game is enhanced further.

Further, if the head-mounted display unit is configured such that, whena panorama image is displayed on the head-mounted display unit and theuser who wears the head-mounted display unit turns its head, a panoramaimage or a virtual space over 360 degrees is displayed on thehead-mounted display unit, then the sense of immersion in the video isfurther enhanced and also the operability of an application such as agame is improved.

SUMMARY Technical Problem

If a head-mounted display unit has a head tracking function in thismanner and the viewpoint or the viewing direction is changed in aninterlocked relationship with the movement of the head of the user togenerate a panorama image, then some latency exists from generation todisplay of the panorama image. Therefore, some displacement occursbetween an assumed direction of the head of the user upon generation ofthe panorama image and the direction of the head of the user at a pointof time at which the panorama image is displayed on the head-mounteddisplay unit. Therefore, the user sometimes falls into drunkensensation.

The present invention has been made in view of such a subject asdescribed above, and it is an object of the present invention to providean image generation apparatus and an image generation method which candisplay a corrected image whose latency after generation to display ofan image is reduced.

Solution to Problem

In order to solve the subject described above, an image generationapparatus according to an aspect of the present invention includes anacquisition unit configured to acquire information relating to at leastone of a viewpoint position and a viewing direction, an image generationunit configured to generate an image using information relating to atleast one of a viewpoint position and a viewing direction acquired at acertain point of time by the acquisition unit, and a correctionprocessing unit configured to receive information relating to at leastone of a viewpoint position and a viewing direction updated at adifferent point of time from the acquisition unit and correct the imagegenerated by the image generation unit using the updated informationrelating to at least one of the viewpoint position and the viewingdirection.

Also another aspect of the present invention is an image generationapparatus. This apparatus is an image generation apparatus incorporatedin a client connected to a server through a network and includes anacquisition unit configured to acquire visual field informationincluding information relating to at least one of a viewpoint positionand a viewing direction, and a correction processing unit configured toreceive, from the server, an image generated using information relatingto at least one of a viewpoint position and a viewing direction acquiredat a certain point of time by the acquisition unit, receive informationrelating to at least one of a viewpoint position and a viewing directionupdated at a different point of time from the acquisition unit andcorrect the received image using the updated information relating to atleast one of the viewpoint position and the viewing direction.

Also a further aspect of the present invention is an image generationapparatus. This apparatus includes an acquisition unit configured toacquire information relating to at least one of a position and arotation of the head of a user who wears a head-mounted display unit, animage generation unit configured to generate an image to be displayed onthe head-mounted display unit using information relating to at least oneof a position and a rotation acquired at a certain point of time by theacquisition unit, and a correction processing unit configured to receiveupdated information relating to at least one of a position and arotation at a different point of time from the acquisition unit andcorrect the image generated by the image generation unit using theupdated information relating to at least one of the position and therotation.

A still further aspect of the present invention is an image generationmethod. This method includes an acquisition step of acquiringinformation relating to at least one of a viewpoint position and aviewing direction, an image generation step of generating an image usinginformation relating to at least one of a viewpoint position and aviewing direction acquired at a certain point of time by the acquisitionstep, and a correction processing step of receiving information relatingto at least one of a viewpoint position and a viewing direction updatedat a different point of time and correcting the image generated by theimage generation step using the updated information relating to at leastone of the viewpoint position and the viewing direction.

Also a yet further aspect of the present invention is an imagegeneration method. This method includes an acquisition step of acquiringinformation relating to at least one of a position and a rotation of thehead of a user who wears a head-mounted display unit, an imagegeneration step of generating an image to be displayed on thehead-mounted display unit using information relating to at least one ofa position and a rotation acquired at a certain point of time by theacquisition step, and a correction processing step of receiving updatedinformation relating to at least one of a position and a rotation at adifferent point of time and correcting the image generated by the imagegeneration step using the updated information relating to the positionand the rotation.

It is to be noted that also an arbitrary combination of the componentsdescribed above and results of conversion of the representation of thepresent invention between a method, an apparatus, a system, a computerprogram, a data structure, a recording medium and so forth are effectiveas modes of the present invention.

Advantageous Effect of Invention

With the present invention, a corrected image whose latency aftergeneration till display of an image is reduced.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an appearance view of a head-mounted display unit.

FIG. 2 is a functional block diagram of the head-mounted display unit.

FIG. 3 is a view depicting a configuration of a panorama imagegeneration system according to an embodiment.

FIG. 4 is a functional block diagram of a panorama image generationapparatus according to the embodiment.

FIG. 5 is a view illustrating a panorama image displayed on thehead-mounted display unit.

FIG. 6 is a view illustrating a reason why a correction process isrequired for a panorama image to be displayed on the head-mounteddisplay unit.

FIG. 7 is a sequence diagram illustrating a conventional imagegeneration process which does not perform a correction process.

FIG. 8 is a sequence diagram illustrating an image generation processwhich involves a correction process of the embodiment.

FIG. 9A is a view illustrating a relationship between a trackingcoordinate system and an HMD coordinate system.

FIG. 9B is a view illustrating a relationship between the HMD coordinatesystem and a visual field coordinate system.

FIG. 10 is a view illustrating a visual field of a user who wears ahead-mounted display unit thereon.

FIG. 11 is a flow chart illustrating a panorama image generation processby the panorama image generation apparatus of FIG. 4 .

FIG. 12 is a view illustrating an image correction process of theembodiment.

FIG. 13 is a view illustrating an image correction process of amodification to the embodiment.

DESCRIPTION OF EMBODIMENTS (First Embodiment)

FIG. 1 is an appearance view of a head-mounted display unit 100. Thehead-mounted display unit 100 includes a main body unit 110, a fronthead portion contact unit 120, and side head portion contact portions130.

The head-mounted display unit 100 is a display apparatus which ismounted on the head of a user such that the user appreciates a stillpicture or a moving picture displayed on a display unit and listens tosound, music or the like outputted from a headphone.

Position information of a user can be measured by a position sensor suchas a GPS (Global Positioning System) built in or externally mounted onthe head-mounted display unit 100. Further, posture information such asa rotational angle or an inclination of the head of the user, who wearsthe head-mounted display unit 100, can be measured by a posture sensorbuilt in or externally mounted on the head-mounted display unit 100.

The main body unit 110 includes the display unit, a position informationacquisition sensor, the posture sensor, a communication apparatus and soforth. The front head portion contact unit 120 and the side head portioncontact portions 130 may have, as optional parts, biological informationacquisition sensors which can measure biological information such as thebody temperature, pulse rate, blood components, sweating, brain wavesand cerebral blood flow of the user.

The head-mounted display unit 100 may further include a camera forpicking up an image of the eyes of the user. By the camera incorporatedin the head-mounted display unit 100, the line of sight, the movement ofthe pupils, a blink of the eyes and so forth of the user can bedetected.

Here, a generation method of an image to be displayed on thehead-mounted display unit 100 is described. The image generation methodof the present embodiment can be applied not only to the head-mounteddisplay unit 100 in a narrow sense but also to a case in which glasses,an eyeglass-type display unit, an eyeglass type camera, a headphone, aheadset (headphone with a microphone), an earphone, an earring, an earcamera, a cap, a cap with a microphone, a hairband or the like ismounted.

FIG. 2 is a functional block diagram of the head-mounted display unit100.

A control unit 10 is a processor which processes and outputs a signalsuch as an image signal or a sensor signal, instructions or data. Aninput interface 20 accepts an operation signal or a setting signal froma touch panel and a touch panel controller and supplies the acceptedsignal to the control unit 10. An output interface 30 receives an imagesignal from the control unit 10 and causes the display unit to displaythe image signal. A backlight 32 supplies backlight to a liquid crystaldisplay unit.

A communication controlling unit 40 transmits data inputted thereto fromthe control unit 10 to the outside by wire or wireless communicationthrough a network adapter 42 or an antenna 44. Further, thecommunication controlling unit 40 receives data from the outside by wireor wireless communication through the network adapter 42 or the antenna44 and outputs the data to the control unit 10.

A storage unit 50 temporarily stores data or a parameter, an operationsignal and so forth processed by the control unit 10.

A GPS unit 60 receives position information from GPS satellites andsupplies the position information to the control unit 10 in accordancewith an operation signal from the control unit 10. A wireless unit 62receives position information from a wireless base station and suppliesthe position information to the control unit 10 in accordance with anoperation signal from the control unit 10.

A posture sensor 64 detects posture information such as a rotationalangle or an inclination of the main body unit 110 of the head-mounteddisplay unit 100. The posture sensor 64 is implemented by a suitablecombination of a gyro sensor, an acceleration sensor, an angularacceleration sensor and so forth.

An external input/output terminal interface 70 is an interface forconnecting a peripheral apparatus such as a USB (Universal Serial bus)controller. An external memory 72 is an external memory such as a flashmemory.

A clock unit 80 sets time information in response to a setting signalfrom the control unit 10 and supplies time data to the control unit 10.

The control unit 10 can supply an image or text data to the outputinterface 30 so that it is displayed on the display unit or can supplyan image or text data to the communication controlling unit 40 so as tobe transmitted to the outside.

FIG. 3 is a view depicting a configuration of a panorama imagegeneration system according to an embodiment. The head-mounted displayunit 100 is connected to a game machine 200 by an interface forconnecting a peripheral apparatus by wireless communication or a USB.The game machine 200 may be connected further to a server through anetwork. In this case, the server may provide an online application of agame or the like, in which a plurality of users can participate throughthe network, to the game machine 200. The head-mounted display unit 100may be connected to a computer or a portable terminal in place of thegame machine 200.

FIG. 4 is a functional block diagram of a panorama image generationapparatus 700 according to the embodiment. FIG. 4 draws a block diagramin which attention is paid to functions. The functional blocks can beimplemented in various forms only from hardware, only from software orfrom a combination of them.

The panorama image generation apparatus 700 is incorporated in the gamemachine 200 to which the head-mounted display unit 100 is connected.However, at least part of functions of the panorama image generationapparatus 700 may be incorporated in the control unit 10 of thehead-mounted display unit 100. Particularly, functions of a correctionprocessing unit 780 hereinafter described may be incorporated in thehead-mounted display unit 100 side. Alternatively, at least part offunctions of the panorama image generation apparatus 700 may beincorporated in the server connected to the game machine 200 through thenetwork.

A zooming instruction acquisition unit 710 acquires a magnification forzooming instructed by a user through the input interface 20 of thehead-mounted display unit 100. The zoom magnification acquired by thezooming instruction acquisition unit 710 is supplied to a sensitivityadjustment unit 720 and a panorama image processing unit 750.

A position and rotation information acquisition unit 730 acquiresinformation relating to the position and the rotation of the head of theuser who wears the head-mounted display unit 100 on the basis ofposition information detected by the GPS unit 60 or a motion sensor ofthe head-mounted display unit 100 and posture information detected bythe posture sensor 64. The position of the head of the user may beacquired by detection of a movement of the head-mounted display unit 100by the camera of the game machine 200.

The position and rotation information acquisition unit 730 acquires theposition and the rotation of the head of the user on the basis of asensitivity instructed by the sensitivity adjustment unit 720. Forexample, if the user turns the neck, then a variation of the angle ofthe head of the user is detected by the posture sensor 64. Thus, thesensitivity adjustment unit 720 instructs the position and rotationinformation acquisition unit 730 to ignore the variation of the detectedangle until the variation of the angle exceeds a predetermined value.

Further, the sensitivity adjustment unit 720 adjusts the sensitivity forangle detection of the head on the basis of the zoom magnificationacquired from the zooming instruction acquisition unit 710. As the zoommagnification increases, the sensitivity for angle detection of the headis lowered. If zooming is performed, then since the field angledecreases, vibration of the display image caused by the swing of thehead can be suppressed by lowering the angle detection sensitivity ofthe head.

As the motion sensor, a combination of at least one of a three-axisgeomagnetic sensor, a three-axis acceleration sensor and a three-axisgyro (angular velocity) sensor may be used to detect forward-backward,leftward-rightward and upward-downward movements of the head of theuser. Alternatively, position information of the head of the user may becombined to improve the sensitivity of motion detection of the head.

A coordinate transformation unit 740 uses the position and the rotationof the head-mounted display unit 100 acquired by the position androtation information acquisition unit 730 to perform coordinatetransformation for generating an image to be displayed on thehead-mounted display unit 100 with a tracking function.

The panorama image processing unit 750 reads out panorama image datafrom a panorama image storage unit 760, generates a panorama image by azoom magnification designated from the zooming instruction acquisitionunit 710 in response to the position and the rotation of thehead-mounted display unit 100 in accordance with the coordinatetransformation by the coordinate transformation unit 740 and providesthe panorama image to the correction processing unit 780. Here, thepanorama image data may be a content of a moving picture or a stillpicture produced in advance or may be rendered computer graphics.

The correction processing unit 780 acquires an updated latest positionand rotation of the head-mounted display unit 100 from the position androtation information acquisition unit 730 and corrects the imagegenerated by the panorama image processing unit 750 using the updatedlatest position and rotation. Details of the correction process arehereinafter described.

The correction processing unit 780 provides an image after corrected toan image provision unit 770. The image provision unit 770 supplies thepanorama image data generated by the panorama image processing unit 750to the head-mounted display unit 100.

FIG. 5 is a view illustrating a panorama image 500 displayed on thehead-mounted display unit 100. When the user is directed to the leftfront with respect to the panorama image 500, an image 510 a included inthe range of a field angle 150 a in the direction of the head-mounteddisplay unit 100 a is displayed, but when the user turns the neck and isdirected to the right front, an image 510 b included in the range of afield angle 150 b in the direction of the head-mounted display unit 100b is displayed.

Since the viewpoint and the viewing direction from and in which thepanorama image displayed on the head-mounted display unit 100 is viewedvary in response to the movement of the head in this manner, the senseof immersion in the panorama image can be raised.

FIG. 6 is a view illustrating a reason why a correction process isrequired for the panorama image to be displayed on the head-mounteddisplay unit 100. If the user turns the neck and is directed to theright front, then the image 510 b included in the range of the fieldangle 150 b in the direction of the head-mounted display unit 100 b isgenerated and displayed on the head-mounted display unit 100. However,at the point of time at which the image 510 b is displayed, the positionand the rotation of the head-mounted display unit 100 b already indicatea change as indicated by reference character 150 c. Therefore, althoughit is originally necessary to display an image which is viewed withinthe range of a field angle 150 c on a head-mounted display unit 100 c,the image generated and displayed actually is an image which is viewedin the range of the field angle 150 b in the direction of thehead-mounted display unit 100 b at a point of time a little earlier. Dueto the displacement by the time difference, an image in a direction alittle displaced from the direction in which the user views is displayedon the head-mounted display unit 100, and the user sometimes feels akind of drunkenness.

In the present embodiment, in order to eliminate this displacement, aprocess for correcting the generated image is performed. First, for thecomparison, a conventional image generation process in which acorrection process is not performed is described with reference to FIG.7 , whereafter a correction process of the present embodiment isdescribed with reference to FIG. 8 .

FIG. 7 is a sequence diagram illustrating a conventional imagegeneration process in which a correction process is not performed.

The panorama image generation apparatus 700 performs preparations forassets such as a three-dimensional object or a texture to acquire aposition p1 and a rotation q1 of the head-mounted display unit 100 attime t1. Concurrently with the asset preparations, the panorama imagegeneration apparatus 700 preforms a process for rendering an image basedon the position p1 and the rotation q1 at time t1. For example, whererendering is performed at the rate of 60 frames/second, approximately 16milliseconds are required for generation of an image of one frame.

The image generated by the panorama image generation apparatus 700 issupplied to the head-mounted display unit 100. The head-mounted displayunit 100 and the panorama image generation apparatus 700 are connectedto each other by wire connection or wireless connection, and a fixedtransmission time is required for the supply of an image from thepanorama image generation apparatus 700 to the head-mounted display unit100. Where the panorama image generation apparatus 700 and thehead-mounted display unit 100 are connected to each other by a network,a network delay occurs.

The head-mounted display unit 100 acquires an image generated by thepanorama image generation apparatus 700 and performs a displayingprocess such as scanning for displaying the image on the panel thereof.A delay is generated by the displaying process, and an image isdisplayed at time t′.

In this manner, after the position p1 and the rotation q1 of thehead-mounted display unit 100 are provided to the panorama imagegeneration apparatus 700 at time t1 until the image is displayed on thepanel of the head-mounted display unit 100 at time t′, a fixed period oftime is required for the rendering, image transmission and displayingprocess, and some latency occurs as depicted in FIG. 7 . Also betweentime t1 at which the position and the rotation of the head-mounteddisplay unit 100 are provided for image generation and time t′ at whichthe image is displayed on the head-mounted display unit 100, the userwho wears the head-mounted display unit 100 migrates or changes theposture. As a result, the user comes to view an image based on theposition and the rotation of the head-mounted display unit 100 in thepast by the time difference Δt=t′−t1, and the user would feel“drunkenness” due to the displacement between the position and rotationon which the displayed image is based and the position and the rotationat present.

FIG. 8 is a sequence diagram illustrating an image generation processwhich involves a correction process of the present embodiment.

The image generation process until the panorama image generationapparatus 700 acquires a position p1 and a rotation q1 at time t1 andperforms asset preparation and rendering processes and then provides agenerated image to the head-mounted display unit 100 is similar to theconventional image generation process of FIG. 7 . In the presentembodiment, at time t2 at which the panorama image generation apparatus700 provides an image to the head-mounted display unit 100, a correctionprocess is performed for the generated image. This correction processmay be performed by any of the head-mounted display unit 100 and thepanorama image generation apparatus 700. Where the head-mounted displayunit 100 has a sufficient processing performance, the head-mounteddisplay unit 100 can perform a correction process. However, if this isnot the case, then the panorama image generation apparatus 700 performsa correction process and supplies an image after the correction to thehead-mounted display unit 100.

In the correction process, information of a position p2 and a rotationq2 of the head-mounted display unit 100 at time t2 at which an image isgenerated is acquired, and the image is corrected on the basis of thedisplacement in position and rotation of the head-mounted display unit100 between time t1 upon starting of rendering and latest time t2. Thehead-mounted display unit 100 performs a display process of thecorrected image to display the image on the panel. Consequently, theapparent latency is reduced to the difference between time t2 and timet′ as depicted in FIG. 11 .

In the following, while the correction process of the present embodimentis described, prerequisite technical matters are described first.

(1) Coordinate System

FIGS. 9A and 9B are views illustrating coordinate systems used by thehead-mounted display unit 100. The head-mounted display unit 100 uses atracking coordinate system 802, an HMD coordinate system 804 and aleftward-rightward visual field coordinate system 806.

FIG. 9A is a view illustrating a relationship between the trackingcoordinate system 802 and the HMD coordinate system 804.

The tracking coordinate system 802 is a coordinate system which makes areference to the position p and the rotation q of the head-mounteddisplay unit 100. The tracking coordinate system 802 may be a Cartesiancoordinate system, and the axes may be selected to any directions andalso the origin may be determined arbitrarily. A coordinate system whichis convenient for the sensors adopted in the head-mounted display unit100 is selected. For example, upon starting of an application such as agame, a user who wears the head-mounted display unit 100 may be causedto assume a reference posture at a reference position, whereupon areference position p0 and a reference rotation q0 of the head-mounteddisplay unit 100 are acquired from sensor information of thehead-mounted display unit 100 to determine the tracking coordinatesystem 802.

Alternatively, the user may have a motion controller 350 of the gamemachine 200 by hand. The user would operate the motion controller 350 ina state in which the head-mounted display unit 100 is worn thereon.Depending upon the game application, the hand may be moved while themotion controller 350 is held or the body may be moved. The motioncontroller 350 includes a three-axis gyro sensor, a three-axisacceleration sensor and a geomagnetism sensor.

A marker 300 is applied to the motion controller 350. The position ofthe marker 300 is detected by the camera connected to the game machine200, and the three-dimensional coordinates of the marker 300 can bespecified accurately together with position information obtained fromthe sensors of the motion controller 350. By setting the trackingcoordinate system 802 of the head-mounted display unit 100 to thethree-dimensional coordinates of the marker 300, the position and theposture of a virtual object which is controlled by the movement of themotion controller 350 and the viewpoint and the viewing direction fromand in which the panorama image 500 controlled by the movement of thehead-mounted display unit 100 is viewed can be processed synchronously.

The HMD coordinate system 804 is a movement coordinate system forrepresenting the position of a device such as the display panel disposedon the head-mounted display unit 100 worn by the user. Although there isno limitation to the origin and the axes, for the convenience ofdescription, the origin is defined so as to be the center of the head ofthe user. Meanwhile, the axes are defined such that, in the state inwhich the user wears the head-mounted display unit 100, the upwarddirection in the upward and downward direction is defined as Y axis; therightward direction from the front is defined as X axis; and thedirection toward this side from the front is defined as Z axis.

The user would move in a real space, change the direction of the body orturn the head in the state in which the head-mounted display unit 100 isworn. Upon starting of an application, the head-mounted display unit 100which is in the reference position p0 and the reference rotation q0moves together with time, and at present, the head-mounted display unit100 is in the position p and the rotation q as depicted in FIG. 9A.

FIG. 9B is a view depicting a relationship between the HMD coordinatesystem 804 and the visual field coordinate system 806.

The HMD coordinate system 804 is a coordinate system wherein the top ofthe head of the user who wears the head-mounted display unit 100 is theorigin while the upward direction is the Y axis; the rightward directionfrom the front is the X axis; and the direction toward this side fromthe front is the Z axis. On the other hand, the visual field coordinatesystem 806 is a coordinate system for determining the direction ofdisplays for the left eye and the right eye. The rightward directionfrom one of the left and right eyes is defined as the X axis; the upwarddirection as the Y axis; and the direction toward this side from thefront as the Z axis.

(2) Coordinate Transformation

In order to display an image on the panel of the head-mounted displayunit 100, three coordinate transformations are interposed. All of thethree coordinate transformations are affine transformations.

First, the image generation coordinate system (in the CG, the cameracoordinate system) is transformed into the tracking coordinate system802. Consequently, a view coordinate system when a real world or avirtual world in which a real or virtual object exists is mapped to thetracking coordinate system which represents the reference position andreference rotation of the head-mounted display unit 100.

Then, the values at present of the position p and the rotation q of thehead-mounted display unit 100 are acquired from the sensor information,and the tracking coordinate system 802 is transformed into the HMDcoordinate system 804. Consequently, the reference position and thereference rotation of the head-mounted display unit 100 are transformedinto the position and the rotation at present.

Finally, the HMD coordinate system 804 is transformed into the visualfield coordinate system 806. This transformation is a transformationwhich depends upon the person having the head-mounted display unit 100worn thereon and the state in which the head-mounted display unit 100 ismounted and is fixed while the head-mounted display unit 100 remainsmounted on the person. Consequently, the HMD coordinate system 804 istransformed into a coordinate system suitable for the eyes of the use ofthe head-mounted display unit 100.

Although, upon the image generation, coordinate transformation fortransforming the tracking coordinate system 802 into the visual fieldcoordinate system 806 via the HMD coordinate system 804 is used, uponthe correction process, coordinate transformation for inverselytransforming the visual field coordinate system 806 into the trackingcoordinate system 802 via the HMD coordinate system 804 is used.

(3) Viewing Angle Data of Head-Mounted Display Unit 100

FIG. 10 is a view illustrating the visual field of the user who wearsthe head-mounted display unit 100. The visual field (field of view)extending vertically and horizontally from the center of the eyes isdefined by four parameters FoVu, FoVb, FoVl and FoVr, and information ofthe viewing angle is defined by an arctangent using the parameters.Since the visual field for the right eye and the visual field for theleft eye are defined similarly to each other, totaling eight parametersare used.

Now, an outline of a procedure for generating an image to be displayedon the head-mounted display unit 100 by the panorama image generationapparatus 700 of the present embodiment is described. Detailedcalculating formulae are hereinafter described.

FIG. 11 is a flow chart illustrating a panorama image generationprocedure by the panorama image generation apparatus 700.

As an initialization process, a coordinate transformation matrix from animage generation coordinate system into a tracking coordinate system isgenerated (S10). This is a work for determining a position and arotation which make a reference for a movement of the head-mounteddisplay unit 100. At a position instructed by the user, an initialposition p0 and an initial rotation q0 are acquired from sensorinformation of the head-mounted display unit 100, and a coordinatetransformation matrix from the image generation coordinate system intothe tracking coordinate system is determined using the acquired values.For example, in order to determine, for example, in a race game, aposture of a person seated in a car in the game as a reference, the useris caused to assume a posture in which it is seated on a chair anddirected forwardly upon starting of a race, and the position and therotation then are determined as references.

From the sensor information of the head-mounted display unit 100, theposition p1 and the rotation q1 of the head-mounted display unit 100 attime t1 are acquired (S12). The head-mounted display unit 100 providesthe position p1 and the rotation q1 at time t1 to the panorama imagegeneration apparatus 700 (S14).

The panorama image generation apparatus 700 performs rendering of animage to be displayed on the display unit of the head-mounted displayunit 100 at the position p1 and the rotation q1 at time t1 (S16).

In particular, the position and rotation information acquisition unit730 acquires the position p1 and the rotation q1 at time t1 from thehead-mounted display unit 100. The coordinate transformation unit 740performs transformation from the image generation coordinate system(camera coordinate system) into the visual field coordinate system 806via the tracking coordinate system 802 and the HMD coordinate system804, and the panorama image processing unit 750 renders a panorama imagefor each of the left and right eyes. Here, upon transformation from thetracking coordinate system 802 to the HMD coordinate system 804, affinetransformation generated from the position p1 and the rotation q1 of thehead-mounted display unit 100 at time t1 acquired by the position androtation information acquisition unit 730 is used. The viewing angle isadjusted to the viewing angle information acquired from the head-mounteddisplay unit 100.

After the rendering at time t1 by the panorama image generationapparatus 700 is completed, the position p2 and the rotation q2 of thehead-mounted display unit 100 at time t2 are acquired from the sensorinformation of the head-mounted display unit 100 (S18). The head-mounteddisplay unit 100 provides the position p2 and the rotation q2 at time t2to the panorama image generation apparatus 700 (S20).

Also while the panorama image generation apparatus 700 is rendering animage, the user who wears the head-mounted display unit 100 changes thedirection, and therefore, the position p2 and the rotation q2 at time t2of the head-mounted display unit 100 are displaced a little from theposition p1 and the rotation q1 at time t1, respectively.

The panorama image generation apparatus 700 performs a process forcorrecting the rendered images in order to absorb the displacement inposition and rotation of the head-mounted display unit 100 between timet1 and time t2 (S22).

In particular, the position and rotation information acquisition unit730 acquires the position p2 and the rotation q2 updated at latest timet2 from the head-mounted display unit 100 and provides the position p2and the rotation q2 to the correction processing unit 780. Thecorrection processing unit 780 further acquires the position p1 and therotation q1 at time t1 of the head-mounted display unit 100, which havebeen utilized by the coordinate transformation unit 740 in order tocreate an affine transformation, from the position and rotationinformation acquisition unit 730. Further, the correction processingunit 780 acquires viewing angle data representative of the renderingfield angle of the display unit of the head-mounted display unit 100.The correction processing unit 780 acquires the generated images fromthe panorama image processing unit 750 and renders a rectangle based onthe position p1 and the rotation q1 at time t1 on the visual fieldcoordinate system with the visual field coordinate system in regard tothe position p2 and the rotation q2 at time t2.

The coordinates of the four vertices of the rectangle on the visualfield coordinate system by the position p1 and the rotation q1 at timet1 are given by

(−FoVl×d, FoVu×d, −d) (−FoVl×d, −FoVb×d, −d) (FoVr×d, FoVu×d, −d)(FoVr×d, −FoVb×d, −d)

(refer to FIG. 11 ). Here, as the value d, the focal distance of theoptical system of the head-mounted display unit 100 is used.

The correction processing unit 780 maps the four vertices of therectangle to the visual field coordinate system in regard to theposition p2 and the rotation q2 at time t2. This is performed by suchcoordinate transformation as described below.

(A) First, the visual field coordinate system is inversely transformedinto the HMD coordinate transform using the values of the position p1and the rotation q1 at time t1, and the HMD coordinate system is furtherinversely transformed into the tracking coordinate system. As a result,the visual field coordinate system as viewed from the position p1 andthe rotation q1 at time t1 is transformed back into the trackingcoordinate system in regard to the reference position p0 and thereference rotation q0 once.

(B) Then, the tracking coordinate system is transformed into the HMDcoordinate system using the values of the position p2 and the rotationq2 at time t2, and the HMD coordinate system is further transformed intothe visual field coordinate system. Consequently, the trackingcoordinate system based on the reference position p0 and the referencerotation q0 is converted into the visual field transform system asviewed from the position p2 and the rotation q2 at time t2.

If the entire coordinate transformations of (A) and (B) are viewed, thenit is regarded that the visual field coordinate system in a case whereit is viewed from the position p1 and the rotation q1 at time t1 isconverted into the visual field coordinate system in another case inwhich it is viewed from the position p2 and the rotation q2 at time t2.

If, to the rectangle in the visual field obtained by the coordinatetransformation in this manner, an image generated assuming the positionp1 and the rotation q1 at time t1 is pasted as a texture, then theresulting image is a corrected image where it is viewed from the visualfield coordinate system in regard to the position p2 and the rotation q2at time t2. The image in which the displacement in position and rotationof the head-mounted display unit 100 appearing between time t1 and timet2 is corrected can be seen in the visual field at present of the user.Thus, the latency by the time difference between time t1 and time t2 isabsorbed, and the “drunkenness” of the user is moderated.

The image provision unit 770 provides the image data corrected by thecorrection processing unit 780 to the head-mounted display unit 100(S24). The panorama image after the correction is displayed on thehead-mounted display unit 100 (S26). Thereafter, the processing returnsto step S12 to repeat the processes at the steps beginning with stepS12.

In the following, the correction process by the correction processingunit 780 is described in detail using mathematical formulae.

As a presumption, it is assumed that information of the position and therotation from absolute references of the head-mounted display unit 100can be provided by the sensors. As the sensors, the GPS unit 60 and theposture sensor 64 of the head-mounted display unit 100 are used.Further, the motion controller 350 may be used. The position p and therotation q vary in response to a movement of the user who wears thehead-mounted display unit 100. Although the head-mounted display unit100 is a rigid body and is not a point, the position p is defined as aposition of one point fixed on the head-mounted display unit 100. In thefollowing description, the fixed one point is referred to as the centralpoint of the head-mounted display unit 100.

An HMD position rotation matrix H is determined in accordance with thefollowing expressions from a translation matrix T and a rotation matrixR using the position p and the rotation q of the head-mounted displayunit 100.

(Expression 1)

H(p,q)=T(−p)×R(q ⁻¹)

p=(tx,ty,tz)

q=(q1,q2,q3,q4)

|q|=1

The position p is a three-dimensional vector, and the rotation q is aquaternion (quaternion). −p is a result of change of the sign of theelements of p to the negative, and q⁻¹ is a conjugate of q (the realpart of q is left as it is while the sign of the imaginary part of q ischanged to the negative).

It is to be noted that the parallel translation by the translationmatrix T(p) is defined by the following expression (2).

$\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & t_{x} \\0 & 1 & 0 & t_{y} \\0 & 0 & 1 & t_{z} \\0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\z \\1\end{pmatrix}}} & \lbrack {{Expression}2} \rbrack\end{matrix}$

The rotation by the rotation matrix R(q) is defined by the followingexpression.

$\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}{q_{0}^{2} + q_{1}^{2} - q_{2}^{2} - q_{3}^{2}} & {2( {{q_{1}q_{2}} - {q_{0}q_{3}}} )} & {2( {{q_{1}q_{3}} + {q_{0}q_{2}}} )} & 0 \\{2( {{q_{1}q_{2}} + {q_{0}q_{3}}} )} & {q_{0}^{2} - q_{1}^{2} + q_{2}^{2} - q_{3}^{2}} & {2( {{q_{2}q_{3}} - {q_{0}q_{1}}} )} & 0 \\{2( {{q_{1}q_{3}} - {q_{0}q_{2}}} )} & {2( {{q_{2}q_{3}} + {q_{0}q_{1}}} )} & {q_{0}^{2} - q_{1}^{2} - q_{2}^{2} + q_{3}^{2}} & 0 \\0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\z \\1\end{pmatrix}}} & \lbrack {{Expression}3} \rbrack\end{matrix}$

A calculation method of three-dimensional rotation and so forth relatingto computer graphics using a quaternion is described in “Introduction toQuaternion for 3D-CG Programmers” (Kogakusha, January, 2004).

In coordinate transformation in general computer graphics, using a worldtransformation matrix W, a camera (view) transformation matrix V and aperspective transformation matrix P, a point (x, y, z, 1) of the worldcoordinate system is transformed into a point (x′, y′, z′, 1) of aprojection coordinate system in accordance with the following expression4.

(Expression 4)

(x′,y′,z′,1)=(x,y,z,1)×W×V×P

In order to display on the display unit of the head-mounted display unit100, the position and the rotation of the head-mounted display unit 100must be tracked to display them on the basis of the position and therotation of the head-mounted display unit 100 at present. Therefore, inimage generation for the head-mounted display unit 100 with a trackingfunction, coordinate transformation in accordance with the expressiongiven below is performed further using a tracking referencetransformation matrix B, an HMD position rotation matrix H and a displayposition matrix D. It is to be noted here that calculation which doesnot involve the correction process described hereinabove is indicated.

(Expression 5)

(x′,y′,z′,1)=(x,y,z,1)×W×V×B×H×D×P

Here, the tracking reference transformation matrix B is a matrix fortransforming the camera (view) coordinate system into the trackingcoordinate system 802 based on the reference position and the referencerotation of the head-mounted display unit 100. The tracking referencetransformation matrix B can be determined as an inverse matrix of theHMD position rotation matrix H generated from sensor information whenthe head-mounted display unit 100 is placed at the reference position p0and the reference rotation q0.

The HMD position rotation matrix H is a matrix for transforming thetracking coordinate system 802 into the HMD coordinate system 804. TheHMD position rotation matrix H(p, q) is generated from the translationmatrix T and the rotation matrix R using the position p and the rotationq acquired from the sensors upon starting of rendering as describedhereinabove.

The HMD display position matrix D is a matrix for transforming the HMDcoordinate system 804 into the visual field coordinate system 806. Bythis matrix, a point of the head-mounted display unit 100 on the HMDcoordinate system 804 whose origin is the top of the head is convertedinto a point on the visual field coordinate system 806 whose origin isthe viewpoint of the user who wears the head-mounted display unit 100.This matrix is determined from the distance from the central point ofthe head-mounted display unit 100 to the position of the eyes of theuser who wears the head-mounted display unit 100 and the rotationalangle by which the direction from the eyes of the user to the center ofthe display unit is inclined with respect to the center axis of thehead-mounted display unit 100. Since the difference of the position ofthe eyes depending upon the person who wears the head-mounted displayunit 100 is sufficiently small, generally the HMD display positionmatrix D is fixed for the device of the head-mounted display unit 100.

It is to be noted that, since the HMD display position matrix relates tothe position of an eye, HMD display position matrices for the left eyeand the right eye exist and are used to generate images for the left eyeand the right eye, respectively. Depending upon the system, a same imageis used for both eyes, and in this case, a single HMD display positionmatrix for both eyes is used.

The perspective transformation matrix P is a matrix for transforming aview coordinate into a perspective coordinate, and in general computergraphics, a field angle is determined freely. However, in thehead-mounted display unit 100, the field angle is adjusted to the fieldangle of the wearing person of the head-mounted display unit 100. Theperspective transformation matrix P is defined as an expression givebelow using the four parameters FoVu, FoVb, FoVl and FoVr indicative ofthe viewing angle of the user.

$\begin{matrix}\begin{pmatrix}\frac{2}{{FoVl} + {FoVr}} & 0 & 0 & \frac{{FoVl} - {FoVr}}{{FoVl} + {FoVr}} \\0 & \frac{2}{{FoVu} + {FoVb}} & 0 & \frac{{- {FoVu}} + {FoVb}}{{FoVu} + {FoVb}} \\0 & 0 & \alpha & \beta \\0 & 0 & {\pm 1} & 0\end{pmatrix} & \lbrack {{Expression}6} \rbrack\end{matrix}$

Here, α and β depend upon clipping of the depth, and ±1 depends uponwhether the coordinate system is for the left hand or for the righthand. Since the field angles for the left eye and the right eye aresometimes different from each other (generally the field angles aresymmetrical to each other in the horizontal direction), the perspectivetransformation matrix P for the left eye and the perspectivetransformation matrix P for the right eye may be different from eachother.

The correction process by the correction processing unit 780 isdescribed. Inputs provided to the correction processing unit 780 includean image generated with regard to the position p1 and the rotation q1 ofthe head-mounted display unit 100 at time t1, the position p1 and therotation q1 used in the HMD position rotation matrix H when the image isgenerated and the updated position p2 and rotation q2 of thehead-mounted display unit 100 at latest time t2.

The correction processing unit 780 generates a corrected image byperforming the following transformation process for a rectangular imageobtained by mapping the image generated with regard to the position p1and the rotation q1 at time t1 on the projection plane. Where thevertices of the rectangular image are represented by (x, y, z, 1), thepositions on the projection coordinate system after correction by (x′,y′, z′, 1) and the correction matrix by C, the convention process can berepresented in the following manner.

(Expression 7)

(x′,y′,z′,1)=(x,y,z,1)×C

Here, the correction matrix C is

C=D ⁻¹ ×H(p1,q1)⁻¹ ×H(p2,q2)×D×P

The four vertices of the rectangular image generated with regard to theposition p1 and the rotation q1 at time t1 are (−FoVl×d, FoVu×d, −d, 1),(−FoVl×d, −FoVb×d, −d, 1), (FoVr×d, FoVu×d, −d, 1) and (FoVr×d, −FoVb×d,−d, 1). As the value of d, for example, the focal distance of theoptical system of the head-mounted display unit 100 is used.

D⁻¹×H(p1, q1)⁻¹ of the front half of the correction matrix C signifiesthat the visual field coordinate system is transformed back into the HMDcoordinate system by the inverse matrix D⁻¹ of the HMD display positionmatrix and the HMD coordinate system is transformed back into thetracking coordinate system by the inverse matrix H(p1, q1)⁻¹ of the HMDposition rotation matrix in which the position p1 and the rotation q1 attime t1 are used. Further, (p2, q2)×D×P of the latter half signifiesthat the tracking coordinate system is transformed into the HMDcoordinate system by the HMD position rotation matrix H(p2, p2) in whichthe updated position p2 and rotation q2 at time t2 are used and then theHMD coordinate system is transformed into the visual field matrix systemby the HMD display position matrix D, whereafter the visual fieldcoordinate system is transformed into the projection coordinate systemby the perspective transformation matrix P finally.

If the position and the rotation of the head-mounted display unit 100 donot vary between time t1 and time t2, namely, if p1=p2 and besides theq1=q2, then since H(p1, q1)⁻¹×H(p2, q2) becomes a unit matrix.Therefore, the correction matrix C becomes equal to the perspectivetransformation matrix P and the correction process is not performed, andonly perspective transformation is performed as given below.

(x′,y′,z′,1)=(x,y,z,1)×P

However, if at least one of the position and the rotation of thehead-mounted display unit 100 varies between time t1 and time t2,namely, if p1≠P2 and/or q1≠q2, then the correction matrix C acts tocorrect the displacement in position and/or rotation of the head-mounteddisplay unit 100 by calculation of H(p1, q1)⁻¹×H(p2, q2).

In this manner, the rectangular image transformed by the correctionmatrix C is a result of correction of the projection image with regardto the position p1 and the rotation q1 at time t1 to the projectionimage with regard to the position p2 and the rotation q2 at time t2.Consequently, even if the position and the rotation of the head-mounteddisplay unit 100 indicate a displacement between time t1 and time t2,the image can be corrected by the image process to absorb thedisplacement.

In the following, several modifications are described.

(Modification 1)

A modification to the correction process of the first embodiment isdescribed with reference to FIGS. 12 and 13 . FIG. 12 is a viewillustrating, for comparison, the image correction process of the firstembodiment. As depicted in the figure, at time t1, the panorama imagegeneration apparatus 700 receives a position p1 and a rotation q1 of thehead-mounted display unit 100 and starts rendering. At time t2, thepanorama image generation apparatus 700 performs a correction processfor a generated image using the latest position p2 and the rotation q2of the head-mounted display unit 100. If it is assumed that the framerate of the panel of the head-mounted display unit 100 is 30frames/second, the panorama image generation apparatus 700 generates aframe at 30 frames/second and performs the correction process and thenprovides an image after the correction to the head-mounted display unit100. It is to be noted that the correction process may be performed onthe head-mounted display unit 100 side.

FIG. 13 is a view illustrating the image correction process of themodification to the first embodiment. While the frame rate of therendering by the panorama image generation apparatus 700 is 30frames/second, if it is assumed that the frame rate of the panel of thehead-mounted display unit 100 is a higher frame rate and is, forexample, 60 frames/second, then, in the present modification, thefrequency of the correction process is raised in accordance with theframe rate of the panel of the head-mounted display unit 100.

As depicted in the figure, a first correction process is performed forthe image generated by the panorama image generation apparatus 700 usingthe position p2 and the rotation q2 of the head-mounted display unit 100at time t2, and the image after the correction is displayed on thehead-mounted display unit 100. Thereafter, a second correction processis performed for the same image using a position p3 and a rotation q3 ofthe head-mounted display unit 100 at time t3, and the image after thecorrection is displayed on the head-mounted display unit 100.Consequently, even if the frame rate of the panorama image generationapparatus 700 is 30 frames/second, the image after the correction isdisplayed at 60 frames/second. In this manner, the frame rate can beraised and lowered by performing the correction process in apredetermined frequency. Particularly where the frame rates of thepanorama image generation apparatus 700 and the head-mounted displayunit 100 are different from each other, a function for frame rateconversion can be provided to the system.

In the illustration of FIG. 13 , the first correction process using theposition p2 and the rotation q2 at time t2 and the second correctionprocess using the position p3 and the rotation q3 at time t3 areperformed for the same image generated by the panorama image generationapparatus 700. As a different method, the second correction process maybe performed for the image after the correction generated by the firstcorrection process.

(Modification 2)

Although the coordinate transformation expression (x′, y′, z′, 1)=(x, y,z, 1)×W×V×B×H×D×P includes multiplication of the tracking referencetransformation matrix B and the HMD position rotation matrix H, thiscalculation can be simplified. A matrix of a result when the trackingreference transformation matrix B and the HMD position rotation matrix Hare multiplied is referred to as normalized HMD matrix N. The trackingreference transformation matrix B can be determined as an inverse matrixof the HMD position rotation matrix H generated from the sensorinformation when the head-mounted display unit 100 is placed at thereference position p0 and the reference rotation q0. Further, the HMDposition rotation matrix H(p, q) is generated from the translationmatrix T and the rotation matrix R using the position p and the rotationq acquired from the sensors upon starting of the rendering. Therefore,where an inverse matrix is represented by Inv(.), the normalized HMDmatrix N is calculated in the following manner.

$\begin{matrix}{N = {B \times {H( {p,q} )}}} \\{= {{Inv}( {{T( {{- p}0} )} \times ( {R( {q0^{- 1}} )} ) \times ( {{T( {- p} )} \times {R( q^{- 1} )}} )} }} \\{= {{{Inv}( {R( {q0^{- 1}} )} )} \times {{Inv}( {T( {{- p}0} )} )} \times {T( {- p} )} \times {R( q^{- `} )}}} \\{= {{R( {q0} )} \times {T( {p0} )} \times {T( {- p} )} \times {R( q^{- 1} )}}} \\{= {{R( {q0} )} \times {T( {{p0} - p} )} \times {R( q^{- 1} )}}} \\{= {{R( {q0} )} \times {R( q^{- 1} )} \times {R(q)} \times {T( {{p0} - p} )} \times {R( q^{- 1} )}}} \\{= {{R( {q^{- 1} \times q0} )} \times {T( {{Rot}( {{{p0} - p},q^{- 1}} )} )}}}\end{matrix}$

Here, Rot(p, q) signifies that the vector p is rotated by the quaternionq. In particular, if the vector p after rotation is represented by p′,then p′=q×p×q⁻¹ is calculated. Rot(p0−p, q⁻¹) signifies to rotate thevector p0−p by the quaternion q⁻¹. With the last transformationexpression given above, it is signified that the normalized HMD matrix Nis translated by Rot(p0−p, q⁻¹) after rotated by the quaternion q⁻¹×q0.

If the calculation amount is evaluated, then if only the number of timesof multiplication is considered, then the translation matrix T involves0 times; the rotation matrix R 16 times; the product of the twoquaternions 16 times; and the product of two rotation matrixes 36 times;and the product of the translation matrix T and the rotation matrix R 0times. If the fourth deformation expression given above is used, thenN=R(q0)×T(p0)×T(−p)×R(q⁻¹), and 16 times of multiplication are requiredfor calculation of the rotation matrix R; 0 times of multiplication isrequired for calculation of R×T; and 36 times of multiplication arerequired for product of two times of the other matrices. Consequently,totaling 16×2+36×2=104 times of multiplication are required. Incontrast, if the last transformation expression given above is used,then N=R(q⁻¹×q0)×T(Rot(p0−p, q⁻¹)), and 16 times of multiplication arerequired for the product of two quaternions and 16 times ofmultiplication are required for calculation of the rotation matrix R.Therefore, two times of calculation of a quaternion are required forcalculation of Rot. Therefore, 16+16+16×2=64 times of multiplication arerequired. In comparison, if the normalized matrix N is calculated by thelast transformation expression given above, then the number of times ofmultiplication decreases from 104 to 64 and the calculation issimplified.

(Modification 3)

The position p and the rotation q of the head-mounted display unit 100can be predicted from sensor information in the past using a Kalmanfilter or the like. The image correction process described above can beused to correct the displacement between the predicted value and anactual value.

A prediction unit for sensor information is provided in the head-mounteddisplay unit 100 or the panorama image generation apparatus 700. Theprediction unit predicts a prediction position pf and a predictionrotation qf at a timing at which an image is to be displayed on thebasis of the movement of the head-mounted display unit 100 in the past.The panorama image generation apparatus 700 generates an image in regardto the prediction position pf and the prediction rotation qf. Thecorrection processing unit 780 receives a position pt at present and arotation qt at present from the sensor information of the head-mounteddisplay unit 100 at the present point of time and performs a correctionprocess for the image using the position pt at present and the rotationqt at present. Consequently, even if the prediction position pf and theprediction rotation qf are displaced from the position pt at present andthe rotation qt at present, respectively, the displacements between themcan be absorbed by the image correction process.

(Modification 4)

Since, in the correction process, an image of a result of renderingperformed once is utilized as a rendering source again to performperspective transformation so as to be suitable for the latest positionand rotation, when the movement amount is great, information in a regionwhich is not rendered as yet is sometimes required. Several methods forcoping with a problem of appearance of a missing portion are describedbelow.

The following methods are available as a method for compensating for amissing portion.

-   -   (1) The missing portion is filled up with a color of pixels on        an outermost periphery.    -   (2) The missing portion is filled up with a single color (for        example, black).    -   (3) A region greater than the outermost periphery is rendered in        advance so that no missing portion appears.

In order to take the method (3), as parameters for providing the fourvertices of a rectangular region to be used in coordinate transformationupon image generation for the head-mounted display unit 100 with atracking function and a transformation process upon a correctionprocess, parameters FoVu′, FoVb′, FoVl′ and FoVr′ representative of arendering field angle are used in place of the field angle parametersFoVu, FoVb, FoVl and FoVr unique to the head-mounted display unit. Here,FoVx′>FoVx′ (here, x=u, b, l, r). It is to be noted that, in theperspective transformation matrix P to be used in a transformationprocess upon the correction process, the angle parameters FoVu, FoVb,FoVl and FoVr unique to the head-mounted display unit are used.

Also a method for controlling the movement amount between two timepoints of time t1 and time t2 so that a missing portion may not appearis available.

(1) The movement amount is limited to a limit value A, and even when themovement amount becomes B (>A) exceeding the limit value A, thecorrection process is performed assuming that the movement amount is A.

(2) When the movement amount exceeds the limit value A, the correctionprocess is not performed, and an error is determined and a processtherefor is performed.

(3) The methods (1) and (2) above are combined such that, when themovement amount exceeds the first limit value A but is equal to or lowerthan the second limit value B (>A), it is determined that the movementamount is A and the correction process is performed, but when themovement amount exceeds the second limit value B, the correction processis not performed and a process is performed determining that an errorhas occurred.

By limiting the movement amount in this manner, the movement amount isincluded within the range of the limit value A, and a regioncorresponding to the movement amount can be rendered in advance so thata missing portion may not appear.

The movement amount can be calculated in the following manner. Thevariation amount qd of the rotation can be calculated by qd=q⁻¹×q0 andthe variation amount pd of the position can be calculated by pd=p0−p.The magnitude of the variation amount qd of the rotation depends uponthe real part, and as the real part decreases, the amount of rotationincreases. Since the expression for calculating the normalized HMDmatrix N described hereinabove is N=R(q⁻¹×q0)×T(Rot(p0−q,q⁻¹))=R(qd)×T(Rot(pd, q⁻¹)), the values of the variation amount qd ofthe rotation and the variation amount pd of the position are calculatedalready in the course of calculation of the normalized HMD matrix N.Since the movement amount is obtained in the process of calculation ofthe normalized HMD matrix N in this manner, there is an advantage thatthe process for analyzing the movement from images to detect a motionvector is not required.

By adopting the method of limiting the movement amount, the renderingfield angle can be fixed, and a system which does not give rise tomissing in image after correction as far as the movement amount does notexceed the limit value can be configured. Where a predicted value of asensor is used to perform rendering as in the case of the modification3, the rendering field angle can be determined depending upon themagnitude of the displacement (error) of the predicted value from thereal value. In this case, if it is predicted that the error is great,then a region in the range in which the error is great is rendered, butif it is predicted that the error is small, then a region of the rangein which the error is small is rendered.

(Modification 5)

In the first embodiment, in order to take both of the translationalmovement and the rotation into consideration, three-dimensionalarithmetic operation is required in the correction process. However, thecorrection process can be simplified if it is assumed that only thetranslational movement is performed without involving the rotation.Since it is necessary to supply only the translational movement, only itis necessary to prepare an image processing device configured from aline buffer of a fixed number of rows and capable of performingtranslational movement. Thus, a less expensive system can be configured.

In particular, the correction process may transform coordinates (0, 0,−d) of the center of an image by the correction matrix C. Only it isnecessary to translationally move the movement in the x and y directionsby the variation amount of x and the variation amount of y determined bythe transformation.

If the user wearing the head-mounted display unit 100 shakes the head ina vertical direction or a horizontal direction, then it can bedetermined that the movement of the head is a vertical or horizontaltranslational movement, and the correction process of the presentmodification can be applied to this case.

(Second Embodiment)

In the description of the first embodiment, a correction process when apanorama image generated by the panorama image generation apparatus 700is displayed on the head-mounted display unit 100 is described. However,in the description of a second embodiment, a correction process when auser performs an operation of a camera for a real or virtual space andan image is displayed on a screen of a computer or a portable apparatusis described. In the following description, description of aconfiguration and operation common to those in the first embodiment areomitted, and the correction process by the correction processing unit780 in the second embodiment is described.

In the first embodiment, since it is assumed that a user wearing thehead-mounted display unit 100 moves in a space, coordinatetransformation for transforming the tracking coordinate system 802 tothe visual field coordinate system 806 via the HMD coordinate system 804is required. This is because, in the first embodiment, the viewpoint andthe viewing direction of the camera must be coordinate-transformed intoa position and a rotation at present of the user wearing thehead-mounted display unit 100. In the second embodiment, since theviewpoint and the viewing direction of the camera operated by a userbecomes the viewpoint and the viewing direction of the viewer of thecomputer or the portable apparatus as they are, the coordinatetransformation required by the head-mounted display unit 100 is notrequired.

The camera transformation matrix V in the second embodiment can bedetermined in accordance with the following expression from thetranslation matrix T and the rotation matrix R using the position(viewpoint) p of the camera and the direction (viewing direction) q ofthe camera.

(Expression 8)

V(p,q)=T(−p)×R(q ⁻¹)

Here, the position p of the camera is a three-dimensional vector, andthe direction q of the camera is a quaternion.

In the second embodiment, coordinate transformation in general computergraphics is used. A point (x, y, z, 1) of the world coordinate system istransformed into a point (x′, y′, z′, 1) of the projection coordinatesystem in accordance with the expression given below using the worldtransformation matrix W, camera (view) transformation matrix V andperspective transformation matrix P. It is to be noted here thatattention is paid to the fact that calculation which does not involve acorrection process is indicated.

(Expression 9)

(x′,y′,z′,1)=(x,y,z,1)×W×V×P

Here, although the perspective transformation matrix P is definedsimilarly as in the first embodiment using the four parameters FoVu,FoVb, FoVl and FoVr indicative of the viewing angle of the user, sincethis relies on the field angle, if the camera is zoomed, then the fieldangle varies and also the perspective transformation matrix P ischanged. The viewing angle parameters at time t1 are represented asFoVu1, FoVb1, FoVl1 and FoVr1 and the viewing angle parameters at timet2 are represented as FoVu2, FoVb2, FoVl2 and FoVr2. The perspectivetransformation matrix P at time t1 is represented as p(t1) and theperspective transformation matrix P at time t2 is represented as P(t2).

The correction process by the correction processing unit 780 in thesecond embodiment is described. Inputs provided to the correctionprocessing unit 780 are the position P1 of the camera at time t1, animage generated in the direction Q1 of the camera, the position P1 ofthe camera and the direction Q1 of the camera used in the cameratransformation matrix V when this image is generated, and the positionP2 of the camera and the direction Q2 of the camera at latest time t2.

The correction processing unit 780 performs the following transformationprocess for a rectangular image in which the image generated with regardto the position p1 and the direction q1 at time t1 is mapped to theprojection plane to generate a corrected image. If a vertex of therectangular image is represented by (x, y, z, 1), a point on theprojection coordinate system after the correction by (x′, y′, z′, 1) andthe correction matrix by C, then the transformation process can berepresented in the following manner.

(Expression 10)

(x′,y′,z′,1)=(x,y,z,1)×C

Here, the correction matrix C is

C=V(p1,q1)⁻¹ ×V(p2,q2)×P(t2)

The four vertices of the rectangular image generated with regard to theposition p1 and the rotation q1 at time t1 are (−FoVl1×d, FoVu1×d, −d,1), (−FoVl1×d, −FoVb1×d, −d, 1), (FoVr1×d, FoVu1×d, −d, 1) and (FoVr1×d,−FoVb1×d, −d, 1). As the value of d, a fixed value of a suitabledistance (for example, ten meters or the like) or statistical data ofthe depth (for example, a median of the depth value of the overall area,an average value of the depth in the proximity of the center of the areaor the like) is used.

V(p1, q1)⁻¹ of the front half of the correction matrix C signifies totransform the camera coordinate system back into the world coordinatesystem by an inverse matrix V(p1, q1)⁻¹ of the camera transformationmatrix which uses the position p1 and the rotation q1 at time t1.Further, V(p2, q2)×P(t2) of the latter half signifies to transform theworld coordinate system into the camera coordinate system by the cameratransformation matrix V(p2, q2) which uses the position p2 and therotation q2 at time t2 and further transform the camera coordinatesystem into the projection coordinate system by the perspectivetransformation matrix P(t2) which uses the field angle at time t2.

If it is assumed that the position and the direction of the camera donot vary between time t1 and time t2, namely, if p1=p2 and q1=q2, thensince V(p1, q1)⁻¹×V(p2, q2) becomes a unit matrix, the correction matrixC becomes equal to the perspective transformation matrix P(t2).Consequently, the correction process is not performed, and onlyperspective transformation which reflects the field angle at time t2 issimply performed as given below.

(x′,y′,z′,1)=(x,y,z,1)×P(t2)

However, if at least one of the position and the rotation of the cameravaries between time t1 and time t2, namely, if p1≠p2 and/or q1≠q2, thenthe correction matrix C acts to correct the displacement in positionand/or direction of the camera by calculation of V(p1, q1)⁻¹×V(p2, q2).Further, by performing transformation with the perspectivetransformation matrix P(t2) finally, also the variation in the fieldangle between time t1 and time t2 is reflected.

The rectangular image transformed by the correction matrix C in thismanner is a result of correction of the projection image with regard tothe position p1, rotation q1 and field angle at time t1 to theprojection image with regard to the updated position p2, rotation q2 andfield angle at time t2. Consequently, even if the position and thedirection of the camera vary or the field angle varies by zoomingbetween time t1 and time t2, the image can be corrected to absorb thedisplacement by the image process.

The second embodiment is advantageous for the configuration of a crowdservice in which a plurality of terminals utilize a service provided byan application server through a network. In this case, the imagegeneration apparatus is provided in the server, and any terminaltransmits information of a camera operation to the server and receivesan image generated by the server through the network. However, since afixed latency is caused by time required for image generation by theserver and time required for network transfer, the camera operation ofthe user cannot be reflected on the real time basis. With the secondembodiment, an image generated on the basis of the position and thedirection of the camera upon image generation can be corrected so as tocoincide with the latest position and direction of the camera upon imagedisplay. Consequently, the real time performance can be providedartificially.

Especially, in a game application, where the user uses a button of agame controller, a touch screen of a portable terminal or the like tochange the viewpoint position and the viewing direction in a virtualspace to perform interaction with a character of a different user in thegame, the real time performance makes a significant factor. In such acase, it is useful to artificially provide the real time performance bythe correction process.

As described above, with the first embodiment and the second embodiment,by correcting an image generated on the basis of a viewpoint positionand a viewing direction at a point of time of image generation using aviewpoint position and a viewing direction at a point of time of imagedisplay, a time difference after the point of time of image generationtill the point of time of image display can be absorbed to reduce theapparent latency.

The present invention has been described in connection with theembodiments. The embodiments are illustrative, and it can be recognizedby those skilled in the art that various modifications are possible inregard to the components and the processes of the embodiments and thatalso such modifications fall within the scope of the present invention.Such modifications are described.

REFERENCE SIGNS LIST

10 Control unit, 20 Input interface, 30 Output interface, 32 Backlight,40 Communication controlling unit, 42 Network adapter, 44 Antenna, 50Storage unit, 60 GPS unit, 62 Wireless unit, 64 Posture sensor, 70External input/output terminal interface, 72 External memory, 80 Clockunit, 100 Head-mounted display unit, 110 Main body unit, 120 Front headportion contact unit, 130 Side head portion contact portion, 200 Gamemachine, 300 Marker, 350 Motion controller, 500 Panorama image, 700Panorama image generation apparatus, 710 Zooming instruction acquisitionunit, 720 Sensitivity adjustment unit, 730 Position and rotationinformation acquisition unit, 740 Coordinate transformation unit, 750Panorama image processing unit, 760 Panorama image storage unit, 770Image provision unit, 780 Correction processing unit

Industrial Applicability

The present invention can be applied to a technology for generating ancorrecting an image.

1. An image generation apparatus, comprising: a processor coupled to amemory storing instructions that when executed by the processor,configure the processor to implement: an acquisition process to acquireinformation relating to at least one of a position and a rotation of acamera; an image generation process to generate an image to be displayedon a display unit in a first frequency lower than a second frequencycorresponding to a frame rate of the display unit using informationrelating to at least one of a position and a rotation acquired at acertain point of time by the acquisition process.
 2. The imagegeneration apparatus according to claim 1, wherein the image generationapparatus is a server.
 3. The image generation apparatus according toclaim 2, wherein the server receives information of the camera operationfrom a terminal coupled to the display unit and sends the generatedimage to the terminal via a network.
 4. The image generation apparatusaccording to claim 1, wherein the camera operation is performed for areal or virtual space.
 5. A display unit, comprising: a processorcoupled to a memory storing instructions that when executed by theprocessor, configure the processor to implement: a correction processingprocess to receive an image to be displayed on the display unitgenerated in a first frequency by an image generation apparatus usinginformation relating to at least one of a position and a rotation of acamera acquired at a certain point of time and correct the imagegenerated by the image generation apparatus using updated informationrelating to at least one of the position and the rotation at a differentpoint of time, wherein: the correction processing process receivesupdated information relating to at least one of a position and arotation in a second frequency corresponding to a frame rate of thedisplay unit and performs multiple times of correction of the imagegenerated by the image generation apparatus in the first frequency usinga plurality of the updated information relating to at least one of theposition and the rotation received in the second frequency higher thanthe first frequency.
 6. The display unit according to claim 5, whereinthe correction processing process receives n (where n is an integerequal to or greater than 2) pieces of the updated information relatingto at least one of the position and the rotation in the second frequencyhigher than the first frequency and performs n times of correction ofthe same image generated by the image generation apparatus in the firstfrequency using the n pieces of the updated information relating to atleast one of the position and the rotation.
 7. The display unitaccording to claim 5, wherein the correction processing process receivesn (where n is an integer equal to or greater than 2) pieces of theupdated information relating to at least one of the position and therotation in the second frequency higher than the first frequency andperforms the first correction of the image generated by the imagegeneration apparatus in the first frequency using the first piece of theupdated information relating to at least one of the position and therotation and then performs the i-th (where i=2, . . . , n) correction ofthe image corrected by the (i−1)-th correction using the i-th piece ofthe updated information relating to at least one of the position and therotation.
 8. The display unit according to claim 5, wherein thecorrection processing process, when the correction of the image causes amissing portion, fills up the missing portion with a color of pixels onan outermost periphery of the image.
 9. The display unit according toclaim 5, wherein the correction processing process, when the correctionof the image causes a missing portion, fills up the missing portion witha predetermined single color.
 10. The display unit according to claim 5,wherein the image generation apparatus renders a region greater than theoutermost periphery of the image in advance so that the correction ofthe image causes no missing portion.
 11. The display unit according toclaim 5, wherein the correction processing process performs thecorrection of the image controlling a movement amount between differenttime points so that the correction of the image causes no missingportion.
 12. The display unit according to claim 11, wherein thecorrection processing process limits the movement amount to apredetermined limit value and when the movement amount exceeds thepredetermined limit value, performs the correction of the image assumingthat the movement amount is the predetermined limit value.
 13. Thedisplay unit according to claim 11, wherein the correction processingprocess, when the movement amount exceeds a predetermined limit value,does not perform the correction of the image but performs an errorprocess.
 14. The display unit according to claim 11, wherein thecorrection processing process sets a first limit value and a secondlimit value greater than the first limit value and when the movementamount exceeds the first limit value but is equal to or smaller than thesecond limit value, performs the correction of the image assuming thatthe movement amount is the first limit value, and when the movementamount exceeds the second limit value, does not perform the correctionof the image but performs an error process.
 15. An image correctionmethod, comprising: a correction processing step for receiving an imageto be displayed on a display unit generated in a first frequency by animage generation apparatus using information relating to at least one ofa position and a rotation of a camera acquired at a certain point oftime and correcting the image generated by the image generationapparatus using updated information relating to at least one of theposition and the rotation at a different point of time, wherein: thecorrection processing step receives updated information relating to atleast one of a position and a rotation in a second frequencycorresponding to a frame rate of the display unit and performs multipletimes of correction of the image generated by the image generationapparatus in the first frequency using a plurality of the updatedinformation relating to at least one of the position and the rotationreceived in the second frequency higher than the first frequency.
 16. Anon-transitory computer-readable medium that stores therein a programcausing a computer to execute an image generation method, the methodcomprising: a correction processing step for receiving an image to bedisplayed on a display unit generated in a first frequency by an imagegeneration apparatus using information relating to at least one of aposition and a rotation of a camera acquired at a certain point of timeand correcting the image generated by the image generation apparatususing updated information relating to at least one of the position andthe rotation at a different point of time, wherein: the correctionprocessing step receives updated information relating to at least one ofa position and a rotation in a second frequency corresponding to a framerate of the display unit and performs multiple times of correction ofthe image generated by the image generation apparatus in the firstfrequency using a plurality of the updated information relating to atleast one of the position and the rotation received in the secondfrequency higher than the first frequency.
 17. A system comprising animage generation apparatus and a display unit, wherein the imagegeneration apparatus comprises: a processor coupled to a memory storinginstructions that when executed by the processor, configure theprocessor to implement: an acquisition process to acquire informationrelating to at least one of a position and a rotation of a camera; animage generation process to generate an image to be displayed on thedisplay unit in a first frequency lower than a second frequencycorresponding to a frame rate of the display unit using informationrelating to at least one of a position and a rotation acquired at acertain point of time by the acquisition process, and wherein thedisplay unit comprises: a processor coupled to a memory storinginstructions that when executed by the processor, configure theprocessor to implement: a correction processing process to receive animage to be displayed on the display unit generated in a first frequencyby an image generation apparatus using information relating to at leastone of a position and a rotation of a camera acquired at a certain pointof time and correct the image generated by the image generationapparatus using updated information relating to at least one of theposition and the rotation at a different point of time, wherein: thecorrection processing process receives updated information relating toat least one of a position and a rotation in a second frequencycorresponding to a frame rate of the display unit and performs multipletimes of correction of the image generated by the image generationapparatus in the first frequency using a plurality of the updatedinformation relating to at least one of the position and the rotationreceived in the second frequency higher than the first frequency.
 18. Animage generation apparatus, comprising: a processor coupled to a memorystoring instructions that when executed by the processor, configure theprocessor to implement: an acquisition process to acquire informationrelating to at least one of a position and a rotation of a camera; animage generation process to generate an image to be displayed on adisplay unit using information relating to at least one of a positionand a rotation acquired at a certain point of time by the acquisitionprocess; and a correction processing process to receive updatedinformation relating to at least one of a position and a rotation at adifferent point of time from the acquisition process and correct theimage generated by the image generation process using the updatedinformation relating to at least one of the position and the rotation,wherein: the image generation process generates the image to bedisplayed on the display unit using the information relating to at leastone of the position and the rotation in a first frequency; and thecorrection processing process receives updated information relating toat least one of a position and a rotation from the acquisition processin a second frequency corresponding to a frame rate of the display unitand performs multiple times of correction of the image generated by theimage generation process in the first frequency using a plurality of theupdated information relating to at least one of the position and therotation received in the second frequency higher than the firstfrequency.